Accessing table (spreadsheet) data#
Packages used#
To use python to access the ERDDAP server directly from your python script or jupyter-notebook, you will need
ERDDAPY
Xarray
netcdf4
matplotlib
folium
Note
The package netcdf4 develop by UNIDATA is not needed in the import part of the python script. However, it is the essential package that support netCDF format output from Xarray. The package matplotlib is also not needed in the import part of the python script. It is the essential package that support quick visualization from Xarray.
In this page, we demonstrate how to extract/download data directly from a ERDDAP server and perform data processing, visualization, and export data in python environment.
Tip
Understanding of the ERDDAP server and what it provides is highly recommended before reading the following intructions.
Import python packages#
# execute the two line when using colab
# !pip install erddapy
# !pip install netcdf4
import xarray as xr
from erddapy import ERDDAP
xarray is used for data processing and netCDF file output.
erddapy is used to access the ERDDAP server.
Both package-webpages have more detail explanation on its full range of functionalities. Here we will mainly focusing on getting the data to be displayer and downloaded.
Access TableDAP type data#
In this demostration, we will be getting the table data of CPS Trawl Life History Haul Catch Data from NOAA NMFS ERDDAP server
Firstly, the way to use the erddapy is to setup the destination ERDDAP server as an object in python through ERDDAP (a python class)
#### access the ERDDAP server
e = ERDDAP(
server="https://coastwatch.pfeg.noaa.gov/erddap/", # The URL that the ERDDAP server has
protocol="tabledap", # The data type (griddap or tabledap)
response="opendap", # different output data type that provided by ERDDAP server
)
Note
Like the comment in the code above, three most important keyword arguments (kwarg) to set for the ERDDAP class are server (The URL that the ERDDAP server is located which has the form of "https://.../erddap/"), protocol (The data type one want to get. It is either "tabledap" or "griddap"), and response (For most general use, set the kwarg as "opendap" to request the data through OPeNDAP Data Access Protocol (DAP) and its projection constraints).
By executing the above code block, we have already setup the connection with the desired ERDDAP server.
To request a specific dataset on the server, we need to know the dataset_id.
The fastest way to get the dataset ID is to go into the data page (e.g. https://coastwatch.pfeg.noaa.gov/erddap/tabledap/FRDCPSTrawlLHHaulCatch.html).
The dataset ID is shown on the second line right after institution.
Tip
One can also get the dataset ID directly from the URL shown above (e.g. https://…/FRDCPSTrawlLHHaulCatch.html).
To set the dataset_id, execute
# set the dataset id name
# ex: https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplAquariusSSS3MonthV5.html
# dataset_id = jplAquariusSSS3MonthV5
e.dataset_id = "FRDCPSTrawlLHHaulCatch"
Download data#
Now, all the setting for downloading the data is complete for this simple example.
All we need to do is to fetch the data from the server to local machine memory.
erddapy has a widely used Xarray backend to support export to xarray object.
Simply execute
ds = e.to_xarray()
The ds object constructed by Xarray is a great way to see the gridded data structure and perform quick visualization, preprocessing, and exporting to netCDF format.
ds
<xarray.Dataset>
Dimensions: (row: 17316)
Coordinates:
latitude (row) float32 ...
longitude (row) float32 ...
time (row) datetime64[ns] ...
Dimensions without coordinates: row
Data variables: (12/15)
cruise (row) float64 ...
ship (row) object ...
haul (row) float64 ...
collection (row) float64 ...
stop_latitude (row) float32 ...
stop_longitude (row) float32 ...
... ...
ship_spd_through_water (row) float32 ...
itis_tsn (row) float64 ...
scientific_name (row) object ...
subsample_count (row) float64 ...
subsample_weight (row) float32 ...
remaining_weight (row) float32 ...
Attributes: (12/38)
acknowledgment: National Oceanic and Atmospheric Administratio...
cdm_data_type: Point
contributor_name: Fisheries Research Division
Conventions: CF-1.6, ACDD-1.3, COARDS
creator_email: swfsc.cps-lht@noaa.gov
creator_name: Coastal Pelagic Species Life History Program
... ...
subsetVariables: cruise, ship, haul, collection,scientific_name...
summary: Weight in kilograms for all species (identifie...
time_coverage_end: 2022-09-29T10:23:00Z
time_coverage_start: 2003-07-09T04:03:00Z
title: CPS Trawl Life History Haul Catch Data
Westernmost_Easting: -134.0793jupyter output cell above shows the coordinates, variables, and related attributes of the datasets and variables.
Visualize data#
To quickly visualize the different variables (with the help of the installed matplotlib package not imported but supporting the plot method in Xarray),
import numpy as np
cruises = np.unique(ds.cruise.data)
cruises
array([200307., 200403., 200404., 200407., 200503., 200504., 200604.,
200704., 200706., 200804., 200807., 200904., 201004., 201010.,
201104., 201204., 201207., 201208., 201304., 201307., 201404.,
201407., 201504., 201507., 201604., 201607., 201704., 201707.,
201807., 201907., 202103., 202107., 202207.])
import matplotlib.pyplot as plt
for cruise in cruises:
yy = ds.where(ds.cruise == cruise,drop=True).stop_latitude.data
xx = ds.where(ds.cruise == cruise,drop=True).stop_longitude.data
plt.plot(xx,yy,'o',markersize=1)
import folium
from folium import plugins
ds.stop_latitude.min().data
lon = (ds.stop_longitude.min().data + ds.stop_longitude.max().data) / 2
lat = (ds.stop_latitude.min().data + ds.stop_latitude.max().data) / 2
fmap = folium.Map(location=[lat, lon], tiles="OpenStreetMap", zoom_start=4)
marker_cluster = plugins.MarkerCluster().add_to(fmap)
for cruise in cruises:
yy = ds.where(ds.cruise == cruise,drop=True).stop_latitude.data
xx = ds.where(ds.cruise == cruise,drop=True).stop_longitude.data
for point in range(0, len(yy)):
folium.Marker([yy[point],xx[point]], popup=f'Cruise #: {cruise:0.0f}').add_to(marker_cluster)
fmap